Focusing on the issue that the Oriented fast and Rotated Brief (ORB) algorithm does not have scale invariance, an improved algorithm based on Speed-Up Robust Feature (SURF) and ORB was proposed. First, the feature points were detected by Hessian matrix, which made the extracted feature points have scale invariance. Second, the feature descriptors were generated by the ORB. Then the K-nearest neighbor algorithm was used for rough matching. Finally, the ratio test, symmetry test, the Least Median Squares (LMedS) theorem was used for purification. When the scale changed, the proposed algorithm's matching precision was improved by 74.3 percentage points than the ORB and matching precision was improved by 4.8 percentage points than the SURF. When the rotation changed, the proposed algorithm's matching precision was improved by 6.6 percentage points than the ORB. The proposed algorithm's matching time was above the SURF, below the ORB. The experimental results show that the improved algorithm not only keeps the rotation invariance of ORB, but also has the scale invariance, and the matching accuracy is improved greatly without decreasing the speed.